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r~^ ' This paper provides simple lower bounds on the number of iterations which is required for successful 

message-passing decoding of some important families of graph-based code ensembles (including low-density 

parity-check codes and variations of repeat-accumulate codes). The transmission of the code ensembles is 

assumed to take place over a binary erasure channel, and the bounds refer to the asymptotic case where we 

ry^ ' let the block length tend to infinity. The simplicity of the bounds derived in this paper stems from the fact 

O . that they are easily evaluated and are expressed in terms of some basic parameters of the ensemble which 

include the fraction of degree-2 variable nodes, the target bit erasure probability and the gap between the 

channel capacity and the design rate of the ensemble. This paper demonstrates that the number of iterations 

^ ' which is required for successful message-passing decoding scales at least like the inverse of the gap (in rate) 

^^O , to capacity, provided that the fraction of degree-2 variable nodes of these turbo-like ensembles does not 

vanish (hence, the number of iterations becomes unbounded as the gap to capacity vanishes). 



Abstract 



Index terms - Accumulate-repeat-accumulate (ARA) codes, binary erasure channel (BEG), 
density evolution (DE), extrinsic information transfer (EXIT) charts, iterative message-passing 
Q I decoding, low-density parity-check (LDPC) codes, stability condition. 
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r\ ; 1 Introduction 

During the last decade, there have been many developments in the construction and analysis of 
low-complexity error-correcting codes which closely approach the Shannon capacity limit of many 
standard communication channels with feasible complexity. These codes are understood to be codes 
defined on graphs, together with the associated iterative decoding algorithms. Graphs serve not 
only to describe the codes themselves, but more importantly, they structure the operation of their 
efficient sub-optimal iterative decoding algorithms. 

Proper design of codes defined on graphs enables to asymptotically achieve the capacity of 
the binary erasure channel (BEG) under iterative message-passing decoding. Gapacity-achieving 
sequences of ensembles of low-density parity-check (LDPG) codes were originally introduced by 
Shokrollahi [5H] and by Luby et al. [T3], and a systematic study of capacity-achieving sequences 
of LDPG ensembles was presented by Oswald and Shokrollahi [18J for the BEG. Analytical bounds 
on the maximal achievable rates of LDPG ensembles were derived by Barak et al. [6J for the 
asymptotic case where the block length tends to infinity; this analysis provides a lower bound on 



the gap between the channel capacity and the achievable rates of LDPC ensembles under iterative 
decoding. The decoding complexity of LDPC codes under iterative message-passing decoding scales 
linearly with the block length, though their encoding complexity is in general super-linear with the 
block length; this motivated the introduction of repeat-accumulate codes and their more recent 
variants (see, e.g., [1], [TO] and [20]) whose encoding and decoding complexities under iterative 
message-passing decoding are both inherently linear with the block length. Due to the simplicity of 
the density evolution analysis for the BEC, suitable constructions of capacity- achieving ensembles 
of variants of repeat-accumulate codes were devised in [10], [TO], [2^ and [25]. All these works 
rely on the density evolution analysis of codes defined on graphs for the BEC, and provide an 
asymptotic analysis which refers to the case where we let the block length of these code ensembles 
tend to infinity. Another innovative coding technique, introduced by Shokrollahi [29], enables to 
achieve the capacity of the BEC with encoding and decoding complexities which scale linearly with 
the block length, and it has the additional pleasing property of achieving the capacity without the 
knowledge of the erasure probability of the channel. 

The performance analysis of finite-length LDPC code ensembles whose transmission takes place 
over the BEC was introduced by Di et al. [H] . This analysis considers sub-optimal iterative message- 
passing decoding as well as optimal maximum- likelihood decoding. In [2], an efficient approach to 
the design of LDPC codes of finite length was introduced by Amraoui et al.; this approach is 
specialized for the BEC, and it enables to design such code ensembles which perform well under 
iterative decoding with a practical constraint on the block length. In [22j . Richardson and Urbanke 
initiated the analysis of the distribution of the number of iterations needed for the decoding of 
LDPC ensembles of finite block length which are communicated over the BEC. 

For general channels, the number of iterations is an important factor in assessing the decoding 
complexity of graph-based codes under iterative message-passing decoding. The second factor 
determining the decoding complexity of such codes is the complexity of the Tanner graph which is 
used to represent the code; this latter quantity, defined as the number of edges in the graph per 
information bit, serves as a measure for the decoding complexity per iteration. 

The extrinsic information transfer (EXIT) charts, pioneered by Stephan ten Brink [30|,l31j. form 
a powerful tool for an efficient design of codes defined on graphs by tracing the convergence behavior 
of their iterative decoders. EXIT charts provide a good approximative engineering tool for tracing 
the convergence behavior of soft-input soft-output iterative decoders; they suggest a simplified 
visualization of the convergence of these decoding algorithms, based on a single parameter which 
represents the exchange of extrinsic information between the constituent decoders. For the BEC, 
the EXIT charts coincide with the density evolution analysis (see |21j ) which is simplified in this 
case to a one-dimensional analysis. 

A numerical approach for the joint optimization of the design rate and decoding complexity 
of LDPC ensembles was provided in [4J; it is assumed there that the transmission of these code 
ensembles takes place over a memoryless binary-input output-symmetric (MBIOS) channel, and 
the analysis refers to the asymptotic case where we let the block length tend to infinity. For the 
simplification of the numerical optimization, a suitable approximation of the number of iterations 
was used in [4J to formulate this joint optimization as a convex optimization problem. Due to the 
efficient tools which currently exist for a numerical solution of convex optimization problems, this 
approach suggests an engineering tool for the design of good LDPC ensembles which possess an 
attractive tradeoff between the decoding complexity and the asymptotic gap to capacity (where 
the block length of these code ensembles is large enough) . This numerical approach however is not 
amenable for drawing rigorous theoretical conclusions on the tradeoff between the number of itera- 
tions and the performance of the code ensembles. A different numerical approach for approximating 
the number of iterations for LDPC ensembles operating over the BEC is addressed in [14J. 



A different approach for characterizing the complexity of iterative decoders was suggested 
by Khandekar and McEhece (see [11^ \T2\ I15j). Their questions and conjectures were related to 
the tradeoff between the asymptotic achievable rates and the complexity under iterative message- 
passing decoding; they initiated a study of the encoding and decoding complexity of graph-based 
codes in terms of the achievable gap (in rate) to capacity. It was conjectured there that for a large 
class of channels, if the design rate of a suitably designed ensemble forms a fraction 1 — e of the 
channel capacity, then the decoding complexity scales like - In - . The logarithmic term in this 
expression was attributed to the graphical complexity (i.e., the decoding complexity per iteration), 
and the number of iterations was conjectured to scale like -. There is one exception: For the BEC, 
the complexity under the iterative message-passing decoding algorithm behaves like In- (see |13j . 
|24j . |25j and [2E])- This is true since the absolute reliability provided by the BEC allows every 
edge in the graph to be used only once during the iterative decoding. Hence, for the BEC, the 
number of iterations performed by the decoder serves mainly to measure the delay in the decoding 
process, while the decoding complexity is closely related to the complexity of the Tanner graph 
which is chosen to represent the code. The graphical complexity required for LDPC and systematic 
irregular repeat-accumulate (IRA) code ensembles to achieve a fraction 1 — e of the capacity of 
a BEC under iterative decoding was studied in [24J and [25j. It was shown in these papers that 
the graphical complexity of these ensembles must scale at least like In-; moreover, some explicit 
constructions were shown to approach the channel capacity with such a scaling of the graphical 
complexity. An additional degree of freedom which is obtained by introducing state nodes in the 
graph (e.g., punctured bits) was exploited in [19] and [20j to construct capacity-achieving ensembles 
of graph-based codes which achieve an improved tradeoff between complexity and achievable rates. 
Surprisingly, these capacity-achieving ensembles under iterative decoding were demonstrated to 
maintain a bounded graphical complexity regardless of the erasure probability of the BEC. A similar 
result of a bounded graphical complexity for capacity-achieving ensembles over the BEC was also 
obtained in [9]. 

This paper provides simple lower bounds on the number of iterations which is required for 
successful message-passing decoding of graph-based code ensembles. The transmission of these 
ensembles is assumed to take place over the BEC, and the bounds refer to the asymptotic case 
where the block length tends to infinity. The simplicity of the bounds derived in this paper stems 
from the fact that they are easily evaluated and are expressed in terms of some basic parameters 
of the considered ensemble; these include the fraction of degree-2 variable nodes, the target bit 
erasure probability and the gap between the channel capacity and the design rate of the ensemble. 
The bounds derived in this paper demonstrate that the number of iterations which is required for 
successful message-passing decoding scales at least like the inverse of the gap (in rate) to capacity, 
provided that the fraction of degree-2 variable nodes of these turbo-like ensembles does not vanish 
(hence, the number of iterations becomes unbounded as the gap to capacity vanishes). The behavior 
of these lower bounds matches well with the experimental results and the conjectures on the number 
of iterations and complexity, as provided by Khandekar and McEliece (see |11) , [12] and [15] ) . Note 
that lower bounds on the number of iterations in terms of the target bit erasure probability can be 
alternatively viewed as lower bounds on the achievable bit erasure probability as a function of the 
number of iterations performed by the decoder. As a result of this, the simple bounds derived in 
this paper provide some insight on the design of stopping criteria for iteratively decoded ensembles 
over the BEC (for other stopping criteria see, e.g., [Hlf25]). 

This paper is structured as follows: Section [5] presents some preliminary background, definitions 
and notation, Section [3] introduces the main results of this paper and discusses some of their 
implications, the proofs of these statements and some further discussions are provided in Section [H 
Finally, Section [5] summarizes this paper. Proofs of some technical statements are relegated to the 
appendices. 



2 Preliminaries 

This section provides preliminary background and introduces notation for the rest of this paper. 

2.1 Graphical Complexity of Codes Defined on Graphs 

As noted in Section [H the decoding complexity of a graph-based code under iterative message- 
passing decoding is closely related to its graphical complexity, which we now define formally. 

Definition 1 (Graphical Complexity). Let C be a binary linear block code of length n and rate 
R, and let Q be an arbitrary representation of C by a Tanner graph. Denote the number of edges 
in Q by E. The graphical complexity of Q is defined as the number of edges in Q per information 
bit of the code C, i.e., A(^) = ^ . 

Note that the graphical complexity depends on the specific Tanner graph which is used to 
represent the code. An analysis of the graphical complexity for some families of graph-based codes 
is provided in P HI [20l IH ES] . 

2.2 Accumulate-Repeat- Accumulate Codes 

Accumulate-repeat-accumulate (ARA) codes form an attractive coding scheme of turbo-like codes 
due to the simplicity of their encoding and decoding (where both scale linearly with the block 
length), and due to their remarkable performance under iterative decoding [l]. By some suitable 
constructions of puncturing patterns, ARA codes with small maximal node degree are presented 
in [T]; these codes perform very well even for short to moderate block lengths, and they suggest 
flexibility in the design of efficient rate-compatible codes operating on the same ARA decoder. 

Ensembles of irregular and systematic ARA codes, which asymptotically achieve the capacity 
of the BEC with bounded graphical complexity, are presented in ^20j. This bounded complexity 
result stays in contrast to LDPC ensembles, which have been shown to require unbounded graphical 
complexity in order to approach channel capacity, even under maximum-likelihood decoding (see 
[24j ) . In this section, we present ensembles of irregular and systematic ARA codes, and give a 
short overview of their encoding and decoding algorithms; this overview is required for the later 
discussion. The material contained in this section is taken from [20l Section II], and is introduced 
here briefly in order to make the paper self-contained. 

From an encoding point of view, ARA codes are viewed as interleaved and serially concatenated 
codes. The encoding of ARA codes is done as follows: first, the information bits are accumulated 
(i.e., differentially encoded), and then the bits are repeated a varying number of times (by an 
irregular repetition code) and interleaved. The interleaved bits are partitioned into disjoint sets 
(whose size is not fixed in general), and the parity of each set of bits is computed (i.e., the bits are 
passed through an irregular single parity-check (SPC) code). Finally, the bits are accumulated a 
second time. A codeword of systematic ARA codes is composed of the information bits and the 
parity bits at the output of the second accumulator. 

Since the iterative decoding algorithm of ARA codes is performed on the appropriate Tanner 
graph (see Fig. [T]), this leads one to view them as sparse-graph codes from a decoding point of view. 



Following the notation in [20j , we refer to the three layers of bit nodes in the Tanner graphs as 
'systematic bits' which form the systematic part of the codeword, 'punctured bits' which correspond 



to the output of the first accumulator and are not a part of the transmitted codeword, and 'code 
bits' which correspond to the output of the second accumulator and form the parity-bits of the 
codeword (see Fig. [T]). Denoting the block length of the code by n and its dimension by k, each 
codeword is composed of k systematic bits and n — k code bits. The two layers of check nodes 
are referred to as 'parity-check 1' nodes and 'parity-check 2' nodes, which correspond to the first 
and the second accumulators of the encoder, respectively. An ensemble of irregular ARA codes is 
defined by the block length n and the degree distributions of the 'punctured bit' and 'parity-check 2' 
nodes. Following the notation in [20], the degree distribution of the 'punctured bit' nodes is given 
by the power series 

oo 

L(x)^^L,x^ (1) 

where Lj designates the fraction of 'punctured bit' nodes whose degree is i. Similarly, the degree 
distribution of the 'parity-check 2' nodes is given by 



R{x) = ^Rix' (2) 



i=l 



where Ri designates the fraction of these nodes whose degree is i. In both cases, degree of a node 
only refers to edges connecting the 'punctured bit' and the 'parity-check 2' layers, without the extra 
two edges which are connected to each of the 'punctured bit' nodes and 'parity-check 2' nodes from 
the accumulators (see Fig. [1]). Considering the distributions from the edge perspective, we let 

oo oo 

A(x)^^A,x*-\ p(x)^^p,x*-i (3) 

1=1 1=1 

designate the degree distributions from the edge perspective; here. A, {pi) designates the fraction of 
edges connecting 'punctured bit' nodes to 'parity-check 2' nodes which are adjacent to 'punctured 
bit' fnaritv-check 2') nodes of deeree i. The desien rate of a svstematic ARA ensemble is eiven bv 






eages connectmg puncturea oit noaes to parity-cnecK z noaes wnicn are aajacent to punctureu 
bit' ('parity-check 2') nodes of degree i. The design rate of a systematic ARA ensemble is given by 
R = -^s_ where 

^ ^ iL, = L'(l) = -^ , an ^ ^ ii?, = R'{1) = -^ (4) 

i / X{t)dt i / p{t)dt 

Jo Jo 

designate the average degrees of the 'punctured bit' and 'parity-check 2' nodes, respectively. 

Iterative decoding of ARA codes is performed by passing messages on the edges of the Tanner 
graph in a layer-by-layer approach. Each decoding iteration starts with messages for the 'systematic 
bit' nodes to the 'parity-check 1' nodes, the latter nodes then use this information to calculate new 
messages to the 'punctured bit' nodes and so the information passes through layers down the graph 
and back up until the iteration ends with messages from the 'punctured bit' nodes to the 'parity- 
check 1' nodes. The final phase of messages from the 'parity-check 1' nodes to the 'systematic bit' 
nodes is omitted since the latter nodes are of degree one and so the outgoing message is not changed 
by incoming information. Assume that the code is transmitted over a BEC with erasure probability 
p. Since the systematic bits receive input from the channel, the probability of erasure in messages 
from the 'systematic bit' nodes to the 'parity-check 1' nodes is equal to p throughout the decoding 
process. For other messages, we denote by xi where i = 0, 1, . . . , 5 the probability of erasure of 
the different message types at decoding iteration number I (where we start counting at zero). The 
variable Xq corresponds to the probability of erasure in message from the 'parity-check 1' nodes 
to the 'punctured bit' nodes, xl tracks the erasure probability of messages from the 'punctured 
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Figure 1: Tanner graph of an irregular and systematic accumulate-repeat-accumulate code. This 
figure is reproduced from [20] . 



bit' nodes to the 'parity-check 2' nodes and so on. The density evolution (DE) equations for the 
decoder based on the Tanner graph in Figured] are given in [20], and we repeat them here: 
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The stability condition for systematic ARA ensembles is derived in [20| Section II. D] and states 
that the fixed point x^ = of the iterative decoding algorithm is stable if and only if 



p'A2 p'(l) + 



2pR'{l) 



< 1. 



(6) 



2.3 Big-O notation 

The terms O, fi and G are widely used in computer science to describe asymptotic relationships 
between functions (for formal definitions see e.g., [33]). In our context, we refer to the gap (in rate) 
to capacity, denoted by e, and discuss in particular the case where < e <C 1 (i.e., sequences of 
capacity-approaching ensembles). Accordingly, we define 



/(e) = 0(^g{e)) means that there are positive constants c and 6, such that < f{e) < c g{e) for all 
< e < <5. 

/(e) = r2(g(e)) means that there are positive constants c and 5, such that < c g{e) < /(e) for all 
< e < <5. 



/(e) = G((7(e)) means that there are positive constants ci, ci and 5, such that < ci g(e) < f{e) < 
C2 g{e) for all < e < 6. 

Note that for all the above definitions, the values of c, ci, C2 and (^ must be fixed for the function 
/ and should not depend on e. 



3 Main Results 

In this section, we present lower bounds on the required number of iterations used by a message- 
passing decoder for code ensembles defined on graphs. The communication is assumed to take place 
over a BEC, and we consider the asymptotic case where the block length of these code ensembles 
tends to infinity. 

Definition 2. Let {Cm} ^i^ be a sequence of code ensembles. Assume a common block length 
(rifn) of the codes in Cm which tends to infinity as m grows. Let the transmission of this sequence 
take place over a BEC with capacity C. The sequence {Cm} is said to achieve a fraction 1 — e of 
the channel capacity under some given decoding algorithm if the asymptotic rate of the codes in Cm 
satisfies R > {1 — e)C and the achievable bit erasure probability under the considered algorithm 
vanishes as m becomes large. 

In the continuation, we consider a standard message-passing decoder for the BEC, and address 
the number of iterations which is required in terms of the achievable fraction of the channel capacity 
under this decoding algorithm. 

Theorem 1. [Lower bound on the number of iterations for LDPC ensembles transmit- 
ted over the BEC] Let {{um, X, p)} p,^ be a sequence of LDPC ensembles whose transmission 
takes place over a BEC with erasure probability p. Assume that this sequence achieves a fraction 
1 — e of the channel capacity under message-passing decoding. Let L2 = L2{e) be the fraction of 
variable nodes of degree 2 for this sequence. In the asymptotic case where the block length tends 
to infinity, let / = l{e,p, Pb) denote the number of iterations which is required to achieve an average 
bit erasure probability P^ over the ensemble. Under the mild condition that Pb < pL2(e), the 
required number of iterations satisfies the lower bound 

l{e,p,Pk)>Y^[y^^L^-y^y ^. (7) 



Corollary 1. Under the assumptions of Theorem [H if the fraction of degree-2 variable nodes stays 
strictly positive as the gap (in rate) to capacity vanishes, i.e., if 

limL2(e) > 

e— >0 

then the number of iterations which is required in order to achieve an average bit erasure probability 
Pb < pL2{e) under iterative message-passing decoding scales at least like the inverse of this gap to 
capacity, i.e., 

l{e,p,P^)=n(^ 



Discussion 1. [Effect of messages' scheduling on the number of iterations] The lower 
bound on the number of iterations as provided in Theorem [1] refers to the flooding schedule where 
in each iteration, all the variable nodes and subsequently all the parity-check nodes send messages 
to their neighbors. Though it is the commonly used scheduling used by iterative message-passing 
decoding algorithms, an alternative scheduling of the messages may provide a faster convergence 
rate for the iterative decoder. As an example, [27] considers the convergence rate of a serial 
scheduling where instead of sending all the messages from the variable nodes to parity-check nodes 
and then all the messages from check nodes to variable nodes, as done in the flooding schedule, 
these two phases are interleaved. Based on the density evolution analysis which applies to the 
asymptotic case of an infinite block length, it is demonstrated in [27j that under some assumptions, 
the required number of iterations for LDPC decoding over the BEC with serial scheduling is reduced 
by a factor of two (as compared to the flooding scheduling). It is noted that the main result of 
Theorem [1] is the introduction of a rigorous and simple lower bound on the number of iterations for 
LDPC ensembles which scales like the reciprocal of the gap between the channel capacity and the 
design rate of the ensemble. Though such a scaling of this bound is proved for the commonly used 
approach of flooding scheduling, it is likely to hold also for other efficient approaches of scheduling. 
It is also noted that this asymptotic scaling of the lower bound on the number of iterations supports 
the conjecture of Khandekar and McEliece [llj . 

Discussion 2. [On the dependence of the bounds on the fraction of degree-2 variable 
nodes] The lower bound on the number of iterations in Theorem [T] becomes trivial when the fraction 
of variable nodes of degree 2 vanishes. Let us focus our attention on sequences of ensembles which 
approach the channel capacity under iterative message-passing decoding (i.e., e — > 0). For the 
BEC, several such sequences have been constructed (see e.g. [131 El])- Asymptotically, as the 
gap to capacity vanishes, all of these sequences known to date satisfy the stability condition with 
equality; this property is known as the flatness condition [28]. In [231 Lemma 5], the asymptotic 
fraction of degree 2 variable nodes for capacity-approaching sequences of LDPC ensembles over 
the BEC is calculated. This lemma states that for such sequences which satisfy the following two 
conditions as the gap to capacity vanishes: 

The stability condition is satisfied with equality (i.e., the fiatness condition holds) 

The limit of the ratio between the standard deviation and the expectation of the right degree exists 
and is finite 

then the asymptotic fraction of degree-2 variable nodes does not vanish. In fact, for various 
sequences of capacity approaching LDPC ensembles known to date (see [131 EHl ESJ), the ratio 
between the standard deviation and the expectation of the right degree-distribution tends to zero; in 
this case, [231 Lemma 5] implies that the fraction of degree-2 variable nodes tends to ^ irrespectively 
of the erasure probability of the BEC, as can be verified directly for these code ensembles. 

Discussion 3. [Concentration of the lower bound] Theorem [T] applies to the required number 
of iterations for achieving an average bit erasure probability Py, where this average is taken over 
the LDPC ensemble whose block length tends to infinity. Although we consider an expectation 
over the LDPC ensemble, note that I is deterministic as it is the smallest integer for which the 
average bit erasure probability does not exceed a fixed value. As shown in the proof (see Section H]), 
the derivation of this lower bound relies on the density evolution technique which addresses the 
average performance of the ensemble. Based on concentration inequalities, it is proved that the 
performance of individual codes from the ensemble concentrates around the average performance 
over the ensemble as we let the block length tend to infinity [21| Appendix C]. In light of this 
concentration result and the use of density evolution in Section H] (which applies to the case of an 



infinite block length), it follows that the lower bound on the number of iterations in Theorem [T] is 
valid with probability 1 for individual codes from the ensemble. This also holds for the ensembles 
of codes defined on graphs considered in Theorems [2] and O 

Discussion 4. [On the number of required iterations for show^ing a mild improvement 
in the erasure probability during the iterative process] Note that for capacity-approaching 
LDPC ensembles, the lower bound on the number of iterations tells us that even for successfully 
starting the iteration process and reducing the bit erasure probability by a factor which is below 
the fraction of degree-2 variable nodes, the required number of iterations already scales like -. 
This is also the behavior of the lower bound on the number of iterations even when the bit erasure 
probability should be made arbitrarily small; this lower bound therefore indicates that for capacity- 
approaching LDPC ensembles, a significant number of the iterations is performed for the starting 
process of the iterative decoding where the bit erasure probability is merely reduced by a factor of 
2 as compared to the erasure probability of the channel (see Discussion [2] as a justification for the 
one-half factor). This conclusion is also well interpreted by the area theorem and the asymptotic 
behavior of the two EXIT curves (for the variable nodes and the parity-check nodes) in the limit 
where e ^ 0; as the gap to capacity vanishes, both curves tend to be a step function jumping 
from to 1 at the origin, so the iterations progress very slowly at the initial stages of the decoding 
process. 

In the asymptotic case where we let the block length tend to infinity and the transmission 
takes place over the BEC, suitable constructions of capacity-achieving systematic ARA ensembles 
enable a fundamentally improved tradeoff between their graphical complexity and their achievable 
gap (in rate) to capacity under iterative decoding (see ^\)- The graphical complexity of these 
systematic ARA ensembles remains bounded (and quite small) as the gap to capacity for these 
ensembles vanishes under iterative decoding; this stays in contrast to un-punctured LDPC code 
ensembles |24] and systematic irregular repeat-accumulate (IRA) ensembles [2Sj whose graphical 
complexity necessarily becomes unbounded as the gap to capacity vanishes (see [201 Table I]). This 
observation raises the question whether the number of iterations which is required to achieve a 
desired bit erasure probability under iterative decoding, can be reduced by using systematic ARA 
ensembles. The following theorem provides a lower bound on the number of iterations required to 
achieve a desired bit erasure probability under message-passing decoding; it shows that similarly 
to the parallel result for LDPC ensembles (see Theorem [T]), the required number of iterations for 
systematic ARA codes scales at least like the inverse of the gap to capacity. 

Theorem 2. [Lower bound on the number of iterations for systematic ARA ensembles 
transmitted over the BEC] Let Urim, ^, p)\ ^r, be a sequence of systematic ARA ensembles 
whose transmission takes place over a BEC with erasure probability p. Assume that this sequence 
achieves a fraction 1 — e of the channel capacity under message-passing decoding. Let L2 = -^2(2) 
be the fraction of 'punctured bit' nodes of degree 2 for this sequence (where the two edges related to 
the accumulator are not taken into account). In the asymptotic case where the block length tends 
to infinity, let / = /(e,p, Pb) designate the required number of iterations to achieve an average bit 

erasure probability P\^ of the systematic bits. Under the mild condition that 1 — < /l < pL2(e), 

the number of iterations satisfies the lower bound 

2 

1 




/(e,p,Pb)>2p(l-e) vVM^-A 1-W1-— -• (8) 



As noted in Section 12.21 systematic ARA codes can be viewed as serially concatenated codes 
where the systematic bits are associated with the outer code. These codes can be therefore decoded 



iteratively by using a turbo-like decoder for interleaved and serially concatenated codes. The 
following proposition states that the lower bound on the number of iterations in Theorem [2] is also 
valid for such an iterative decoder. 

Proposition 1. [Lower bound on the number of iterations for systematic ARA codes 
under turbo-like decoding] Under the assumptions and notation of Theorem [21 the lower bound 
on the number of iterations in ([8]) is valid also when the decoding is performed by a turbo-like 
decoder for uniformly interleaved and serially concatenated codes. 



The reader is referred to [Appendix I for a detailed proof. The following theorem which refers to 



irregular repeat-accumulate (IRA) ensembles is proved in a conceptually similar way to the proof 
of Theorem [2j 

Theorem 3. [Lower bound on the number of iterations for IRA ensembles transmit- 
ted over the BEC] Let {('T-m; A, p)}^ j^ be a sequence of (systematic or non-systematic) IRA 
ensembles whose transmission takes place over a BEC with erasure probability p. Assume that 
this sequence achieves a fraction 1 — e of the channel capacity under message-passing decoding. 
Let L2 = L2{e) be the fraction of 'information bit' nodes of degree 2 for this sequence. In the 
asymptotic case where the block length tends to infinity, let / = l{s,p,Pb) designate the required 
number of iterations to achieve an average bit erasure probability Pb of the information bits. For 
systematic codes, if P^ < pL2{£), then the number of iterations satisfies the lower bound 

2 1 



Ke,p,Py,) > 2(1 - e) [^/pL2{e) - y^j -. (9) 



For non-systematic codes, if P^ < -^2(^)1 then 



2 1 



/(e,p,Pb)>2(l-e)(/L2M- /flj -• (10) 



4 Derivation of the Bounds on the Number of Iterations 

4.1 Proof of Theorem [T] 

Let jx'"}, j^ designate the expected fraction of erasures in messages from the variable nodes to 
the check nodes at the Tth iteration of the message-passing decoding algorithm (where we start 
counting at Z = 0). Prom density evolution, in the asymptotic case where the block length tends to 
infinity, x^^' is given by the recursive equation 

a;('+i)=pA(l-p(l-x('))) , /gN (11) 

with the initial condition 

x(°) = p (12) 

where p designates the erasure probability of the BEC . Considering a sequence of { {um , A , p) } LDP C 
ensembles where we let the block length rim tend to infinity, the average bit erasure probability 
after the Tth iteration is given by 

Pi'^=pL{l-p{l-x^'^)) (13) 
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where L designates the common left degree distribution of the ensembles from the node perspective. 
Since the function f{x) = p\{l — p{l — x)) is monotonically increasing, Eqs. ()lip ~ (ll3p imply that 
an average bit erasure probability of Pb is attainable under iterative message-passing decoding if 
and only if 

p\{l- p{l-x)) <x, VxG(x*,p] (14) 

where x* is the unique solution of 



P^,=pL[l-p{l-x*)). 



Let us define the functions 



c{x) = 1- p{l -x), 



v{x) 



A-1 
1 



< X < p 
p < x <\ 



(15) 



From the condition in (J14p . an average bit erasure probability of P\^ is attained if and only if 
c(x) < v{x) for all x S (x*,p]. Since we assume that vanishing bit erasure probability is achievable 
under message-passing decoding, it follows that c{x) < v{x) for all x £ {0,p]. Figure [2] shows a plot 
of the functions c(x) and v{x) for an ensemble of LDPC codes which achieves vanishing bit erasure 
probability under iterative decoding as the block length tends to infinity. The horizontal and vertical 
lines, labeled {^ij^ppj and {i^ij^pj^, respectively, are used to track the expected fraction of erased 
messages from the variable nodes to the parity-check nodes at each iteration of the message-passing 
decoding algorithm. From (jlip and (J12p . the expected fraction of erased left to right messages in 
the rth decoding iteration (where we start counting at zero) is equal to the x value at the left tip 
of the horizontal line hi. The right-angled triangles shaded in gray will be used later in the proof. 

The first step in the proof of Theorem [1] is calculating the area bounded by the curves c{x) and 
v{x). This is done in the following lemma which is based on the area theorem for the BEC [3]. 

Lemma 1. 

fl f-y E> 

(t'(x) — c(x))dx = (16) 



/o «L 

where C = 1 — p \s the capacity of the BEC, R is the design rate of the ensemble, and cl is the 
average left degree of the ensemble. 



Proof. The definitions of the functions v and c in (J15p imply that 



/ (f (x) — c(x))dx = / Al — )dx+/ Idx— / c(x)dx 
Jo Jo \P J Jp Jo 

= p X-^{s)ds + l-p- {l-p{l-x))dx 
Jo Jo 

= p ll — A(x)dx j+1— p— 1+ / p{x)dx 



1 .1 

p{x)dx — p / A(x)dx 
Jo 

( \ 



A(x)dx 



ll p(x)dx 
/q A(x)dx 



C-R 



Sf \ \-R 



\-C 



where (a) follows by substituting x = A(s) and applying integration by parts. 



D 
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CD 

o 



CD 

^_ 

to 

to 



slope=(pA,J ^ 



slope=(pX2) '' 



slope=(pA.2)" 



slope=(pA,2) ^ 

v(x) = ?i"''(x/p) 

xe[0,p] ^3 / / > 



c(x) = 1-p(1-x) 





slope = p'(1) 



slope = p'(1) 



Left to Right Message Erasure Probability 



Figure 2: Plot of the functions c{x) and v{x) for an ensemble of LDPC codes which achieves 
vanishing bit erasure probability under iterative message-passing decoding when communicated 
over a BEC whose erasure probability is equal to p. The horizontal and vertical lines track the 
evolution of the expected fraction of erasure messages from the variable nodes to the check nodes 
at each iteration of the message-passing decoding algorithm. 

Let us consider the two sets of right-angled triangles shown in two shades of gray in Figure [2j 
The set of triangles which are shaded in dark gray are defined so that one of the legs of triangle 
number i (counting from right to left and starting at zero) is the vertical line Vi, and the slope of 
the hypotenuse is equal to c'(0) = p'(l). Since c{x) is concave for all x E [0, 1], these triangles are 
guaranteed to be above the curve of the function c. Since the slope of the hypotenuse is p'(l), the 
area of the i'th triangle in this set is 



A 



Vi 



p'{l)) 2p'{l) 



(17) 



where \vi\ is the length of Vi. We now turn to consider the second set of triangles, which are shaded 
in light gray. Note that the function \{x) is monotonically increasing and convex in [0, 1] and also 
that A(0) = and A(l) = 1. This implies that A~^ is concave in [0, 1] and therefore v{x) is concave 
in [0,p]. The triangles shaded in light gray are defined so that one of the legs of triangle number 
i (again, counting from the right and starting at zero) is the vertical line Vi and the slope of the 
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hypotenuse is given by 

.'(„) = i(A-.)'(0) = ^ = ^ 

where the second equahty follows since A(0) = 0. The concavity of v{x) in [0,p] guarantees that 
these triangles are below the curve of the of function v. The area of the i'th triangle in this second 
set of triangles is given by 

Bi = 2 bil i\vi\ph) = ^ • (18) 

Since v{x) is monotonically increasing with x, the dark-shaded triangles lie below the curve of the 
function v. Similarly, the monotonicity of c{x) implies that the light-shaded triangles are above 
the curve of the function c. Hence, both sets of triangles form a subset of the domain bounded by 
the curves of c(x) and v{x). By their definitions, the i'th dark triangle is on the right of Vi, and 
the i'th light triangle lies to the left of Vi; therefore, the triangles do not overlap. Combining (fD 
p8|) and the fact that the triangles do not overlap, and applying Lemma [H we get 

= / (v(x) — c{x))dx 

«L Jo 

oo 

> ^(A + Bi) 

i=0 

1 

> - 



2 Vp'(1) 



i-i 



PA2 J^b.P (19) 



i=0 



where I is an arbitrary natural number. Since we assume that the bit erasure probability vanishes 
under iterative message-passing decoding, the stability condition implies that 

Substituting i^ and R = {I - e)C in (^ gives 

i-i 
Ce>aLpA2 ^bd^. (21) 

1=0 

The definition of hi and vi in Figure E] implies that for an arbitrary iteration / 

I 



l-p(l-x(')) = c(x(')) = l-^ 



\Vi\ . 
i=0 

Substituting the last equality in (fT^ yields that the average bit erasure probability after iteration 
number I — 1 can be expressed as 

Pt'^=pL(l-Y,\vA . (22) 

Let I designate the number of iterations required to achieve an average bit erasure probability P^ 
over the ensemble (where we let the block length tend to infinity), i.e., I is the smallest integer which 
satisfies P^ < -ft, since we start counting at ^ = 0. Although we consider an expectation over 
the LDPC ensemble, note that / is deterministic as it is the smallest integer for which the average 
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bit erasure probability does not exceed Pb- Since L is monotonically increasing, ([22]) provides a 
lower bound on X]«=o 1^*1 °^ ^^^ form 



i=0 

From the Cauchy-Schwartz inequality, we get 



EN.--'(^) 



(23) 



'i-i \ 2 i-i i-i i~i 



En <EiEi-^i' = ^Ei-^i'- (24) 



\i=0 / j=0 j=0 j=0 

Combining the above inequality with (|21|) and (|23|) gives the inequality 



which provides the following lower bound on the number of iterations I: 

,>!^Mz^;(iir. p., 

(l-p)e 

To continue the proof, we derive a lower bound on 1 — L^^(x) for x G (0, 1). Since the fraction of 
variable nodes of degree i is non-negative for alH = 2, 3, . . ., we have 

L(x) = ^ LiX^ > Lax^ x > 0. 

i 

Substituting t = L{x) gives 

t>L2-{L-^{t)f, VtG(0,l) 

which is transformed into the following lower bound on 1 — L^^ix): 

l-L-\x)>l- .1^, VxG(0,l). (26) 

Under the assumption -^ < L2, substituting (l26]) in (|25]) gives 

/ / — ^ ^ 

ttLpAsf/L^- J^ 

I > - ^ ^ 



L2il-p)e 

0^X2 {VpL2 - VK)' 



L2{l-p)e 
The lower bound in d?]) is obtained by substituting the equality L2 = ^2^ into ([2 



(27) 



Taking the limit where the average bit erasure probability tends to zero on both sides of ([7]) 
gives the following lower bound on the number of iterations: 



\ — p e 
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4.2 Proof of Theorem H 

We begin the proof by considering the expected fraction of erasure messages from the 'punctured 
bit' nodes to the 'parity-check 2' nodes (see Fig. [1]). The following lemma provides a lower bound 
on the expected fraction of erasures in the Tth decoding iteration in terms of this expected fraction 
at the preceding iteration. 

Lemma 2. Let (n, \,p) be an ensemble of systematic ARA codes whose transmission takes place 
over a BEC with erasure probability p. Then, in the limit where the block length tends to infinity, 
the expected fraction of erasure messages from the 'punctured bit' nodes to the 'parity-check 2' 
nodes at the /'th iteration satisfies 

xf^>A(l-p(l-xS'-^M, / = 1,2,... (28) 



where the tilted degree distributions A and p are given as follows (see f2I^ ) : 

2 



and L and R designate the degree distributions of the ARA ensemble from the node perspective. 

Proof. See Appendix IILAI D 

From Fig. [H it can be readily verified that the probabilities xq and xi for erasure messages at 
iteration no. zero are equal to 1, i.e., 

xf = xf ) = 1. (31) 

Let us look at the RHS of ()28p as a function of x, and observe that it is monotonically increasing 
over the interval [0, 1] . Let us compare the performance of a systematic ARA ensemble whose 
degree distributions are (A, p) with an LDPC ensemble whose degree distributions are given by 
(A,p) (see ([29|) and ([30]) ) under iterative message-passing decoding. Given the initial condition 
x\ = 1, the following conclusion is obtained by recursively applying Lemma O For any iteration, 
the erasure probability for messages delivered from 'punctured bit' nodes to 'parity-check 2' nodes 
of the ARA ensemble (see Fig. [T]) is lower bounded by the erasure probability of the left-to-right 
messages of the LDPC ensemble; this holds even if the a-priori information from the BEC is not 
used by the iterative decoder of the LDPC ensemble (note that the coefficient of A in the RHS of 
()28|) is equal to one). Note that unless the fraction of 'parity-check 2' nodes of degree 1 is strictly 
positive (i.e., Ri > 0), the iterative decoding cannot be initiated for both ensembles (unless some 
the values of some 'punctured bits' of the systematic ARA ensemble are known, as in [20]). Hence, 
the comparison above between the ARA and LDPC ensembles is of interest under the assumption 
that i?i > 0; this property is implied by the assumption of vanishing bit erasure probability for the 
systematic ARA ensemble under iterative message-passing decoding. 

In [20] Section H.C.2], a technique called 'graph reduction' is introduced. This technique trans- 
forms the Tanner graph of a systematic ARA ensemble, transmitted over a BEC whose erasure 
probability is p, into a Tanner graph of an equivalent LDPC ensemble (where this equivalence 
holds in the asymptotic case where the block length tends to infinity). The variable and parity- 
check nodes of the equivalent LDPC code evolve from the 'punctured bit' and 'parity-check 2' nodes 
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of the ARA ensemble, respectively, and their degree distributions (from the edge perspective) are 
given by A and p, respectively. It is also shown in [20] that A and p are legitimate degree distribution 
functions, i.e., all the derivatives at zero are non-negative and A(l) = p{l) = 1. As shown in j2(Jt 
Eqs. (9)-(12)], the left and right degree distributions of the equivalent LDPC ensemble from the 
node perspective are given, respectively, by 



L{x) = ^, = ^ r Irr ^ (32) 

'0 



and 








Let P^ designate the average erasure probability of the systematic bits after the /'th decoding 
iteration (where we start counting at / = 0). For LDPC ensembles, a simple relationship between 
the erasure probability of the code bits and the erasure probability of the left-to-right messages 
at the rth decoding iteration is given in ()13p . For systematic ARA ensembles, a similar, though 
less direct, relationship exists between the erasure probability of the systematic bits after the /'th 
decoding iteration and x\ ; this relationship is presented in the following lemma. 

Lemma 3. Let (n. A, p) be an ensemble of systematic ARA codes whose transmission takes place 
over a BEC with erasure probability p. Then, in the asymptotic case where the block length tends 
to infinity, the average erasure probability of the systematic bits after the Tth decoding iteration, 
P^ , satisfies the inequality 




i-yi-^>^(i-?(i-^rjj (34) 

where p and L are defined in (J3U|) and (|32p , respectively (similarly to their definitions in 

Proof. See Appendix IILBI D 

Remark 1. We note that when P^ is very small, the LHS of (I34p satisfies 



p(0 p(0 
V p 2p 

so p4p takes a similar form to (llSp which refers to the erasure probability of LDPC ensembles. 



Consider the number of iterations required for the message-passing decoder, operating on the 
Tanner graphs of the systematic ARA ensemble, to achieve a desired bit erasure probability Pb- 
Combining Lemmas [2] and [3l and the initial condition in (j3ip . a lower bound on this number of 
iterations can be deduced. More explicitly, it is lower bounded by the number of iterations which is 

required to achieve a bit erasure probability of 1 — < /l — -^ for the LDPC ensemble whose degree 

distributions are (A,/?) and where the erasure probability of the BEC is equal to 1. It is therefore 
tempting to apply the lower bound on the number of iterations in Theorem[Tl which refers to LDPC 
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ensembles, as a lower bound on the number of iterations for the ARA ensemble. Unfortunately, 
the LDPC ensemble with the tilted pair of degree distributions (A, p) is transmitted over a BEC 
whose erasure probability is 1, so the channel capacity is equal to zero and the multiplicative gap 
to capacity is meaningless. This prevents a direct use of Theorem [U however, the continuation of 
the proof follows similar lines in the proof of Theorem [TJ 

Let X* denote the unique solution in [0, 1] of the equation 

(35) 

From (j28p . (j3ip and (|34p. a necessary condition for achieving a bit erasure probability Pb of the 
systematic bits is that 

A(l-p(l-2;)) <x, VxG(x*,1]. (36) 

In the limit where the fixed point of the iterative decoding process is attained, the inequalities 
in (|28p. (|3ip and (j34p are replaced by equalities; hence, (|36p also forms a sufficient condition. 
Analogously to the case of LDPC ensembles, as in the proof of Theorem [U we define the functions 

c{x) = 1 — />(1 — x) and v{x) = \~ (x) . (37) 

Due to the monotonicity of A in [0, 1], the necessary and sufficient condition for attaining an erasure 
probability P^ of the systematic bits in (j36p can be rewritten as 

c{x) < v{x) , Vx G (x*, 1] . 

Since we assume that the sequence of ensembles asymptotically achieves vanishing bit erasure 
probability under message-passing decoding, it follows that 

c(x) < v{x) , Vx G (0, 1] . 

The next step in the proof is calculating the area of the domain bounded by the curves c(x) and 
v{x). This is done in the following lemma which is analogous to Lemma [TJ 

Lemma 4. 

f\v{x)-c{x))dx= ^-J^ (38) 



where v and c are introduced in ()37p . C = 1 — p is the capacity of the BEC, R is the design rate of 
the systematic ARA ensemble, and or is defined in (j3|) and it designates the average degree of the 
'parity-check 2' nodes when the two edges related to the lower accumulator in Fig. [T] are not taken 
into account. 

Proof. The definitions of the functions v and c in (j37p yield that 

/ ('u(x) — c(x))dx = X (x)dx — 1 + / p{l — x)dx 

Jo Jo Jo 

1 — / A(x)dx \ — I + p{x)dx 

p{x)dx — / A(x)dx (39) 

Jo 

where the second equality is obtained via integration by parts (note that A(0) =0 and A(l) = 1). 
From ([32]) . we get 

[ A(x)dx = J— = -^ = ^ (40) 

Jo U(l) L'{1) OL 
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(see also [201 Eq. (23)]) where ol is defined in ^ and designates the average degree of the 'punctured 
bit' nodes in the Tanner graph (see Fig. [1]) when the two edges, related to the upper accumulator 
in Fig. [H are not taken into account. Similarly, (|33|) gives 

pix)dx = J— = \^ = ^—^ (41) 

R'{1) R'{^) «R 

(see also [20l Eq. (24)]). Substituting (gOl) and dH]) into ([39]) gives 

-I 
{v{x) — c{x))dx = 

(a) 



i - 


V P 


flR flL 


1 
OR 


-K^) 


1 


1 l-R-p 


OR l-R 


C 


-R 



(1 - R) an 



(42) 



where (a) follows since the design rate of the systematic ARA ensemble is given hy R = ^ ""^ 
(this equality follows directly from Fig. [T]). n 

To continue the proof, we consider a plot similar to the one in Figure [2] with the exception that 
c{x) and v{x) are replaced by c{x) and v{x), respectively. Note that in this case the horizontal line 
ho is reduced to the point (1, 1). Consider the two sets of gray-shaded right-angled triangles. The 
triangles shaded in dark gray are defined so that the height of triangle number i (counting from 
right to left and starting at zero) is the vertical line Vi and the slope of their hypotenuse is equal 
to c'(O) = /o'(l). Since c(x) is concave, these triangles form a subset of the domain bounded by the 
curves c(x) and v{x). The area of the i'th triangle in this set is given by 



M 



2 ' " \p'{l)J 2p'{l) 



where \vi\ is the length of Vi. The second set of right-angled triangles, which are shaded in light 
gray, are also defined so that the height of the i'th triangle (counting from right to left and starting 
at zero) is the vertical line Vi, but the triangle lies to the left of Vi and the slope of its hypotenuse 
is equal to 

where the second equality follows since A(0) = and the third equality follows from the definition 
of A in (j29p . Since A is monotonically increasing and convex over the interval [0, 1] and it satisfies 
A(0) = and A(l) = 1, then it follows that v{x) = X~^{x) is concave over this interval. Hence, the 
triangles shaded in light gray also form a subset of the domain bounded by the curves c(x) and 
v{x). The area of the i'th light-gray triangle is given by 

^i = 2 l^«l (l^»l^ -^2) = 

Applying Lemma U] and the fact that the triangles in both sets do not overlap, we get 

''"'' >if^+P^A.lVN^ (43) 



{l~R)an-2\p'{l) ^ V^ 
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where I is an arbitrary natural number. Since the sequence of ensembles asymptotically achieves 
vanishing bit erasure probability under iterative message-passing decoding, the stability condition 
for systematic ARA codes (see © or equivalently pUl Eq. (14)]) implies that 

where the last equality follows from (I30p . Substituting ()44p in ()43|) gives 

n^'m >P^A,^N^ (45) 

Let x^' denote the x value of the left tip of the horizontal line hi. The value of x^' satisfies the 
recursive equation 

xC+i) = a(i-p(1-x('))), V/gN (46) 



with x^^' = 1. As was explained above (immediately following Lemma [2j), from ()28p . (I3ip . and the 
monotonicity of the function f{x) = A(l — p(l — x)) over the interval [0, 1], we get that x^'^' < x\ 
for I £ N. The definition of hi and vi in Figure [2] implies that 



I 



1 - p(l - x(')) = %(')) = 1 - X] l^il • (4'^) 



j=0 

Starting from (j34p and applying the monotonicity of L and p gives 



P, 



(i-i) 



1-yi-^ > L(l-^(l-xl' 



(i-i) 



> Ln-p{i-x 



(/-I) 



i=0 



where the last equality follows from (j47p . Since L is strictly monotonically increasing in [0, 1], then 



l-i / I p(z-i)- 



^K|>l-L-i(^l-yi-^j. (48) 

Applying the Cauchy-Schwartz inequality (as in (HH)) to the RHS of (|15|) . we get 



.1=0 



> E^L.z-'(..M.!tl 



P 
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where the last inequahty follows from (|48p . Since the design rate R is assumed to be a fraction 
1 — e of the capacity of the BEC, the above inequality gives 



p2A2(l-i2)aR (l-Z-Ml-Wl-5^ 



Ce> 



I 



where I is an arbitrary natural number. Let I designate the number of iterations required to achieve 
an average bit erasure probability P\y of the systematic bits, i.e., / is the smallest integer which 
satisfies Pj^ < -Pb (since we start counting the iterations at / = 0). Note that I is deterministic 
since it refers to the smallest number of iterations required to achieve a desired average bit erasure 
probability over the ensemble. From the inequality above and the monotonicity of L, we obtain 
that 

p'M (1 - R) an (l - Z-i (l - Jl-^ 



I 
which provides a lower bound on the number of iterations of the form 



/ > 



P'A2(1 - 


-R)aK (l 


-Z-.(i 


-yi. 


p J J 




-e)aL (l- 


Ce 






p'A2(l- 


V- 


p )) 



e 



(49) 



where the last equality follows since ^ = j^ (see Fig. [T]) and R = {\ — e)C. To continue the 

proof, we derive a lower bound on 1 — L~'^{x). Following the same steps which lead to p6p gives 
the inequality 

1-L"^(x) > 1- /3^, Vx>0 (50) 

V ^2 



where ()32p implies that 



~ L"(0) pL"{Q) 

^2 = — 7. — = :: =P^2- (51) 



Under the assumption that 1 — ^/1 — -j^ < pL2, substituting ([50]) and ([5T]) in (|19]) gives 



pA2(l-e)aL (VpL^-\l-Jl-^ 

I > ^ p '— ■ (52) 

Lo e 



Finally, the lower bound on the number of iterations in ([8]) follows from (|52|) by substituting 

L2 - —2—. 

Considering the case where Pb — *■ on both sides of ([5]) gives 

/(£,p,Pb-0)>2/(l-£)M£l. 
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5 Summary and Conclusions 

In this paper, we consider the number of iterations which is required for successful message-passing 
decoding of code ensembles defined on graphs. In the considered setting, we let the block length 
of these ensembles tend to infinity, and the transmission takes place over a binary erasure channel 
(BEC). 

In order to study the decoding complexity of these code ensembles under iterative decoding, one 
needs also to take into account the graphical complexity of the Tanner graphs of these code ensem- 
bles. For the BEC, this graphical complexity is closely related to the total number of operations 
performed by the iterative decoder. For various families of code ensembles, Table [U compares the 
number of iterations and the graphical complexity which are required to achieve a given fraction 
1 — e (where e can be made arbitrarily small) of the capacity of a BEC with vanishing bit erasure 
probability. The results in Table [T] are based on lower bounds and some achievability results which 
are related to the graphical complexity of various families of code ensembles defined on graphs (see 
[191 l20l ^^ EH]); the results related to the number of iterations are based on the lower bounds 
derived in this paper. 



Code 
family 


Number of decoding iterations 
as function of e 


Graphical complexity 
as function of e 


LDPC 


n{l) (Theorem d]) 


e (Ini) [24, Theorems 2.1 and 2.3] 


Systematic IRA 


n{l) (Theorem EI) 


e fin ^) [25l Theorems 1 and 2] 


Non-systematic IRA 


n{l) (Theorem EI) 


6(1) [19J 


Systematic ARA 


n{l) (Theorem [2D 


6(1) [20J 



Table 1: Number of iterations and graphical complexity required to achieve a fraction 1 — e of the 
capacity of a BEC with vanishing bit erasure probability under iterative message-passing decoding. 

Theorems [THSJ demonstrate that for various attractive families of code ensembles (including 
low-density parity-check (LDPC) codes, systematic and non-systematic irregular rep eat- accumulate 
(IRA) codes, and accumulate-repeat-accumulate (ARA) codes), the number of iterations which is 
required to achieve a desired bit erasure probability scales at least like the inverse of the gap between 
the channel capacity and the design rate of the ensemble. This conclusion holds provided that the 
fraction of degree-2 variable nodes in the Tanner graph does not tend to zero as the gap to capacity 
vanishes (where under mild conditions, this property is satisfied for sequences of capacity-achieving 
LDPC ensembles, see |23t Lemma 5]). 

When the graphical complexity of these families of ensembles is considered, the results are less 
homogenous. More explicitly, assume a sequence of LDPC codes (or ensembles) whose block length 
tends to infinity, and consider the case where their transmission takes place over a memoryless 
binary-input output-symmetric channel. It follows from [241 Theorem 2.1] that if a fraction 1 — e 
of the capacity of this channel is achieved with vanishing bit error (erasure) probability under ML 
decoding (or any sub-optimal decoding algorithm), then the graphical complexity of an arbitrary 
representation of the codes using bipartite graphs scales at least like In ^ . For systematic IRA codes 
which are transmitted over the BEC and decoded by a standard iterative message-passing decoder, 
a similar result on their graphical complexity is obtained in [251 Theorem 1]. In [24[ Theorem 2.3], 
the lower bound on the graphical complexity of LDPC ensembles is achieved for the BEC (up to a 
small additive constant), even under iterative message-passing decoding, by the right-regular LDPC 
ensembles of Shokrollahi [28]. Similarly, [251 Theorem 2] presents an achievability result of this form 
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for ensembles of systematic IRA codes transmitted over the BEC; the graphical complexity of these 
ensembles scales logarithmically with - . For ensembles of non-systematic IRA and systematic ARA 
codes, however, the addition of state nodes in their standard representation by Tanner graphs allows 
to achieve an improved tradeoff between the gap to capacity and the graphical complexity; suitable 
constructions of such ensembles enable to approach the capacity of the BEC with vanishing bit 
erasure probability under iterative decoding while maintaining a hounded graphical complexity (see 
|19j and [2Qj)- We note that the ensembles in [20] have the additional advantage of being systematic, 
which allows a simple decoding of the information bits. 

The lower bounds on the number of iterations in Theorems [TH3] become trivial when the fraction 
of degree-2 variable nodes vanishes. As noted in Discussion [21 for all known capacity-approaching 
sequences of LDPC ensembles, this fraction tends to ^ as the gap to capacity vanishes. For some 
ensembles of capacity approaching systematic ARA codes presented in [20], the fraction of degree-2 
'punctured bit' nodes (as introduced in Fig. [T]) is defined to be zero (see J2D| Table I]). For these 
ensembles, the lower bound on the number of iterations in Theorem [2] is ineffective. However, this 
is mainly a result of our focus on the derivation of simple lower bounds on the number of iterations 
which do not depend on the full characterization of the degree distributions of the code ensembles. 
Following the proofs of Theorems [1] and [21 and focusing on the case where the fraction of degree-2 
variable nodes vanishes, it is possible to derive lower bounds on the number of iterations which 
are not trivial even in this case; these bounds, however, require the knowledge of the entire degree 
distribution of the examined ensembles. 

The simple lower bounds on the number of iterations of graph-based ensembles, as derived in 
this paper, scale like the inverse of the gap in rate to capacity and also depend on the target bit 
erasure probability. The behavior of these lower bounds matches well with the experimental results 
and the conjectures on the number of iterations and complexity, as provided by Khandekar and 
McEhece (see [11], [12] and [l5]). In [121 Theorem 3.5], it was stated that for LDPC and IRA 
ensembles which achieve a fraction 1 — e of the channel capacity of a BEC with a target bit erasure 
probability of Pb under iterative message-passing decoding, the number of iterations grows like 
O (^) . In light of the outline of the proof of this statement, as suggested in [121 P- 71], it implicitly 
assumes that the flatness condition is satisfied for these code ensembles and also that the target 
bit erasure probability vanishes; under these assumptions, the reasoning suggested by Khandekar 
in [121 Section 3.6] supports the behavior of the lower bounds which are derived in this paper. 

The matching condition for generalized extrinsic information transfer (GEXIT) curves serves to 
conjecture in [l6l Section XI] that the number of iterations scales like the inverse of the achievable 
gap in rate to capacity (see also [TTl p. 92]); this conjecture refers to LDPC ensembles whose trans- 
mission takes place over a general memoryless binary-input output-symmetric (MBIOS) channel. 
Focusing on the BEC, the derivation of the lower bounds on the number of iterations (see Section d]) 
makes the heuristic reasoning of this scaling rigorous. It also extends the bounds to various graph- 
based code ensembles (e.g., IRA and ARA ensembles) under iterative message-passing decoding, 
and makes them universal for the BEC in the sense that they are expressed in terms of some basic 
parameters of the ensembles which include the fraction of degree-2 variable nodes, the target bit 
erasure probability and the asymptotic gap between the channel capacity and the design rate of 
the ensemble (but the bounds here do not depend explicitly on the degree distributions of the code 
ensembles). An interesting and challenging direction which calls for further research is to extend 
these lower bounds on the number of iterations for general MBIOS channels; as suggested in [16[ 
Section XI] , a consequence of the matching condition for GEXIT curves has the potential to lead 
to such lower bounds on the number of iterations which also scale like the inverse of the gap to 
capacity for general MBIOS channels. 
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Appendices 
Appendix I Proof of Proposition [1] 

We begin the proof by considering an iterative decoder of systematic ARA codes by viewing them as 
interleaved and serially concatenated codes. The outer code of the systematic ARA code consists of 
the first accumulator which operates on the systematic bits (see the upper zigzag in Fig. [T]), followed 
by the irregular repetition code. The inner code consists of the irregular SPC code, followed by the 
second accumulator (see the lower zigzag in Fig. [T]). These two constituent codes are joined by an 
interleaver which permutes the repeated bits at the output of the outer code before they are used 
as input to the inner encoder; for the considered ARA ensemble, we assume that the interleaver 
is chosen uniformly at random over all interleavers of the appropriate length. The turbo-like 
decoding algorithm is based on iterating extrinsic information between bitwise MAP decoders of 
the two constituent codes (see e.g., [7]). Each decoding iteration begins with an extrinsic bitwise 
MAP decoding for each non-systematic output bit of the outer code (these are the bits which 
serve as input to the inner code) based on the information regarding these bits received from the 
extrinsic bitwise MAP decoder of the inner code in the previous iteration and the information on the 
systematic bits received from the communication channel. In the second stage of the iteration, this 
information is passed from the outer decoder to an extrinsic bitwise MAP decoder of the inner code 
and is used as a-priori knowledge for decoding the input bits of the inner code. A Tanner graph for 
turbo-like decoding of systematic ARA codes is presented in Figured Considering the asymptotic 
case where the block length tends to infinity, we denote the probability of erasure messages from 
the outer decoder to the inner decoder and vice versa at the Tth decoding iteration by Xq and 
x\ , respectively. Keeping in line with the notation in the proofs of Theorems [1] and [21 we begin 
counting the iterations at / = 0. Since there is no a-priori information regarding the non-systematic 
output bits of the outer decoder (which are permuted to form the input bits of the inner decoder, 
as shown in Fig. [3j) we have 

4-1) = x[-^^ = 1. (I.l) 

We now turn to calculate the erasure probability x^ in an extrinsic bitwise MAP decoding of 
non-systematic output bits of the outer code, given that the a-priori erasure probability of these 
bits is x^ . To this end, we consider the Tanner graph of the outer code, shown in the top box of 
Figure El We note that this Tanner graph contains no cycles, and therefore bitwise MAP decoding 
of this code can be performed by using the standard iterative message-passing decoding algorithm 
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Figure 3: Tanner graph of a systematic accumulate-repeat-accumulate (ARA) code for turbo-like 
decoding as an interleaved and serially concatenated code. 



until a fixed-point is reached. In such a decoder which operates on the Tanner graph of the outer 
code, messages are transferred between the 'punctured bit' and the 'parity-check 1' nodes of the 
graph. Let us denote by xq^o{x) the probability of erasure in messages from the 'punctured bit' 
nodes to the 'parity-check 1' nodes at the fixed point of the iterative decoding algorithm, when 
the a-priori erasure probability of the output bits is x. Similarly, we denote by 2:1^0(2;) the erasure 
probability in messages from the 'parity-check 1' nodes to the 'punctured bit' nodes at the fixed 
point, where x is the a-priori erasure probability of the non-systematic output bits. Based on the 
structure of the Tanner graph, we have 



and 



xq,o{x) = xlo(x) • L{x) 



Xi,o(x) = 1 - (1 - p) (1 - Xq^o{x)) 



(L2) 



(1.3) 



where L is defined in ([T]) and it forms the degree distribution of the 'punctured bit' nodes from the 
node perspective, and p denotes the erasure probability of the BEC. Substituting (JL2]) into ()I.3jl 
gives 



a;i,o(2;) 



P 



(1.4) 



l-{l-p)L{x) ' 
Therefore, the structure of the Tanner graph of the outer code implies that the erasure probability 



M 



in messages from the outer decoder to the inner decoder at iteration number I of the turbo-like 
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decoding algorithm is given by 



.« = (x,,(xf-)))^(.f-)) 



P 



= A(xf-'>) (L5) 

where the second equahty reUes on (|L4p . and A is the tilted degree distribution which results from 
graph reduction (see (f29]l ). We now employ a similar technique to calculate the erasure probability 
x\ in an extrinsic bitwise MAP decoding of input bits of the inner code, given that the a-priori 
erasure probability of these bits is Xq . Since the Tanner of the inner code is also cycle-free (see the 
lower box in Figure [3]), extrinsic bitwise MAP decoding can be done by using the iterative decoder 
operating on the Tanner graph of the inner code. We denote by xq^[{x) the erasure probability of 
messages from the 'parity check 2' nodes to the 'code bit' nodes at the fixed point of the iterative 
decoding algorithm when x is the a-priori erasure probability of the input bits. Similarly, xi^i(x) 
designates the erasure probability of messages from the 'code bit' nodes to the 'parity check 2' 
nodes at the fixed point of the decoding algorithm, when x is the a-priori erasure probability of the 
input bits. The structure of the Tanner graph implies that 

xo,i(x) = 1 - (1 - xi,i(x))i?(l - x) (1.6) 

and 

xi,i(x) =p2;o,i(x) (1.7) 

where R is defined in ([2]). Substituting (JL6J) into (|I.7j) gives 

'^^''^'^^" l-pR{l-x) ■ ^^-^^ 

Therefore, the erasure probability x\ in messages from the inner decoder to the outer decoder at 
iteration number I of the turbo-like decoding algorithm is given by 

x« = l-(l-xi,i(xi')))%(l-x(')) 

' l-pii(l-xK ' '^ ' 

\ ^ 



l-pR{l-x\l>) 



p{l - x\i>) 



= l-p{l- xP) (1.9) 

where the second equality relies on O.SP , and p is the tilted degree distribution resulting from graph 
reduction (see (fHUj) ). Combining p.ip . (|I.5p and (|I.9p gives 

xi°)=A(4-^))=A(l) = l, 

xP = x(l-p{l-x^^~^'^)) , /gN. (I.IO) 



Observing the proof of Theorem [21 we note that Xq = x*^'' for all I = 0, 1, . . ., where is the x*^^) 
value at the left tip of the horizontal line hi in Figure [2] (see Eq. (I46p on page [T9l) . 
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Let P^ designate the average erasure probability of the systematic bits at the end of the /'th 
iteration of the turbo-hke decoder. From the definition of the turbo-hke decoding algorithm, P^ is 
the erasure probability of bitwise MAP decoding for the input bits to the outer code, given that the 
a-priori erasure probability of the output bits of this code is given by x^ . 
of the Tanner graph of the outer code in Figure [3l we get 



Based of the structure 



P 



m 



p 



{l - Xo,o{x?))'' 



(I.ll) 



where 2:0,0(2^) in the fixed point erasure probability of messages from the 'punctured bit' nodes to 
the 'parity-check 1' nodes in the case that the a-priori erasure probability of the non-systematic 
output bits of the code is x. Substituting ()I.3p in 0.2^ gives 



a;o,o(a;) 



pL{x) 



l-{l-p)Lix) 
Substituting the above equality into p. lip , we have 



P 



(0 



P 



P 



1 



pL[x\ 



W\ 



i-{i-p)L{x';>) 



1- L(x 



(0 



1-1-^(1-^(1 



M 



where the second equality follows from the definition of L in 
()I.9|) . Using simple algebra, the above expression gives 



1 



P 



(0 



P 



Lh-p(i 



„(0 



and the third equality relies on 



(1.12) 



Hence, the lower bound on the average erasure probability of the systematic bits at the end of 
the rth iteration of the standard iterative decoder for ARA codes in Lemma [3] is satisfied (with 
equality) also for the turbo-like decoder. 

Let / designate the required number of iterations for the turbo-like decoder to achieve an average 
erasure probability Pb of the systematic bits. Since we start counting the iterations at zero, p.l2p 
implies that / is the smallest natural number which satisfies 



P 



>l(i-p{i 



.C-i) 



However, this is exactly the quantity for which we calculated the lower bound in the proof of 
Theorem [2] (see Lemmas [2] and [3] and Eq. (I3ip ). Therefore, we conclude that the lower bound 
on the number of iterations (/) in Theorem [2] holds also when the considered turbo- like decoding 
algorithm is employed to decode the systematic ARA codes as interleaved and serially concatenated 
codes. 
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Appendix II 



II. A Proof of Lemma [2] 



The proof of Lemma [2] is based on the DE equations in ([5]) for systematic ARA ensembles. From 
the DE equations for X2 and x\\ we have 



.(0 



(0 



= p 

> p 



l-R l-x 



„(0 



l-x 



{i-i) 



l-R{l-x 



„(0 



l-x 



m 



.(0 ;. 



where the inequahty fohows since the decoding process does not add erasures, so x\ is monotoni- 
cally decreasing with I (for i = 0, 1, . . . , 5). This gives 



1 — X3 < 1 — p 



1- R{l-x 



.(0 



l-x 



(0 



and 



l-x?< 



1 — p 



pR{ 



l-x 



(0 



Substituting (JILII) into the DE equation for X4 (see ([5])) gives 



(0 



.(0 



\-xlTp(l-x? 



(II.l) 



> 1 



1 — p 



i-pRli-x];' 

where p is defined in (|30]l . From ([5]) , we get 



p[l-x\ 



(0 



.W 



\ 4 



(/-I) 



i-xr^M(i-p) 



L X 



„(0 



> 



l-{l-xf){l-p)\L(xf 



(11.2) 



where the inequahty follows from the monotonicity of {x^ }. Solving for 1 — x\ gives 



l-xf< 



I- L X 



M 



,(0 



l-{l-p)L{x, 
Substituting (|II.3p into the DE equation for Xq in ([5|) , we have 



(11.3) 



M 



> 1 



l-x 



(1-p) 



C-i) 



)(1-P) 



l-L X 



,a-i) 



l-(l-p)L(^X4 



(/-I) 



l-(l-p)L X 



„('-!) 
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Substituting the inequality above into the DE equation for x\ gives 



X 



(0 



4T^ur' 



> 



P 



l-(l-p)L X, 



Ax 



n-i)\ I •■v-4 



.C-i) 



Ax 



J'-i) 



(11.4) 



where A is defined in (j29|) . Finally (|28|) follows from pi.2p and pi.4p and the nionotonicity of A 
over the interval [0, 1]. 



II. B Proof of Lemma [3] 



From the structure of the Tanner graph of systematic ARA codes (see Fig. [T]) and the DE equation 
for Xg in ([5]) we get 

2" 



H 



(0 



p 



p 



l-(l-xf))' 



1 - ( 1 - xPl (xf 



The DE equation ^ for xf ^ and ([29]) imply that 



„(0 



X 



(0 



Ax 



„(^-i) 



(0 2 
Xj^ p 



Ax 



„a-i) 



> 



1-(1- 


-p)L(.r')] 




\^ 


p 


\ 



l-{l-p)L(x 



„a-i) 



(II.5) 



where the last inequality follows from ()II.4p . Taking the square root on both sides of the above 
inequality gives 

(11.6) 



Xq ^ 



P 



l-(l-p)L X, 



.a-i) 



Substituting ([lL6l) in (jlLS]) . we get 



pf > p 



> p 



1- 1 



1- 1 



pL (X4 



„(0 



(i-p)l(xJ-^)) 



pL(x« 



l-(l-p)L X 



.(0 



(11.7) 
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where the second inequahty above follows since the decoding process does not add erasures so 
^4 ^ ^4 ) ^iid from the monotonicity of L over [0, 1]. Applying the definition of L in (j32p to 
the RHS of (jlLTl) gives 



P^ > p 



1-(1-L(xf''' 

l-L{l-p(x^lA)]\ (IL8) 



> P<1 

where the last inequality follows from 01. 2^ . Finally, (I34p follows directly from (lILSh 
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